Robots Can Wear Multiple Hats in the Computer Science Curriculum at Liberal Arts Colleges

Faculty at liberal arts colleges are often challenged to offer a quality education to their students, complete with opportunities for undergraduate research. To guard against a curriculum that is too theoretical, students want to see applications of their course work and tangible results of their efforts. Like all computer science educators, we want to attract students to our discipline. The use of robotics can often be part of the answer in each of these realms. Undergraduate Research Projects Centre College, like many good liberal arts institutions, is anxious to provide its students with opportunities to engage in undergraduate research. Robotics provides a venue where students can make interesting contributions and where an ongoing project can be maintained. In 2004, enabling funds from a professorship were used to purchase a Sony AIBO and a grant from the Kentucky Space Grant Consortium provided stipends for students involved in the project. These resources were available because machine learning and robotics were of interest to the funding agency. A sophomore and two junior level students were involved in the project over the course of two summers and one academic year. The project was centered around the topic of machine learning, specifically reinforcement learning, and the software package pyro (http://pyrorobotics.org/) was selected as the means of communication between the computer and the AIBO. Because Centre computer science students learn Python as their first language and then Java, use of pyro enabled them to get started on the project more quickly than if they had had to learn C++ first. While this robot has a large number of sensors the work was almost entirely involved with the vision system which depends on a camera mounted in its nose. Vision processing software allowed us to set filters to detect the presence of the bright pink of a ball in the image produced by the camera. The earliest projects involved the ____________ Copyright © 2007, American Association for Artificial Intelligence (www.aaai.org). All rights reserved use of reinforcement learning to teach AIBO to move his head so as to locate the pink ball. The results were noticeable. In one experiment, both the ball and the robot always began in the same position. In this case the average number of steps the robot took to locate the ball decreased with training from 29.9 to 5.58. There were similar results for random initial positions of the robot head. With a small data set, initial path averages were as high 33.14 steps but decreased to less than 8.0 with training. The complexity of the problem increased substantially when locomotion was added to the mix. Now the goal was not only for the robot to locate the ball visually but also to move toward it. When the size of the blob exceeded a threshold value of 750 pixels, the experiment declared he was close enough and the number of steps was recorded. AIBO wagged his tail to announce success. In performing these experiments, one has to make decisions about the sizes of various parameters: how much should the head be moved or tilted, how fast should the robot move forward or turn. Based on ideas contained in the literature (Kohl and Stone 2004) an experiment was devised to find optimal values for six of the parameters. The policy gradient reinforcement learning described there was quite successful in the present setting. Beginning with a base policy which, on average, required 196 steps to locate the ball, the process of generating base policies, testing, and adjusting the policies in the direction of maximum improvement yielded approximately a fifty per cent improvement in performance. (Soenneker, 2006). Many of the experiments dealt with temporal difference learning and the problem of estimating the action value function, ( ) , Q s a π , which is the value of taking action a in state s under policy π . The students only worked with action value functions which were implemented as a table. However, they experimented with several representations of the state, trying to capture as much information as possible while maintaining a reasonable table size. Inspired by a another paper in the literature (Smart and Kaelbling, 1998) they worked on ways to provide initial settings for this array so that the robot did not start with a tabula rasa but instead had some guidance in making early decisions. The result was a program which could be run either in a “teacher” mode or a “learning” mode. In the teaching mode, the choice of action was dictated by the program but based on the rewards or punishments received, the action value function was updated as though the robot had made the decision. Once the program was switched to learning mode, the robot chose its actions based on the contents of the table, slowly improving the estimates depending on the rewards and punishments resulting from its choices. Of course, some of the time the action was chosen randomly to encourage exploration for even better choices than had been tried in the past. ( , Q s a π ) Results from these experiments show that the average number of steps per episode fell from over 350 to less that 250 as training progressed. Observations The kinds of experiments that were carried out by these students gave them a lot of freedom to pose and test hypotheses. In the course of our work, we tried a variety of state descriptions, different types of reward and punishment schemes, and variations in the types of questions asked. Since the students were not required to understand all the mathematical details, the work was accessible to them. One student submitted an abstract on the results of this work to an undergraduate research contest and presented a poster session at a regional meeting. There are a great many more things that can be explored – including the possibility of implementing the entire thing without the use of the pyro software which does add a layer between the programmer and the hardware. The introduction of eligibility traces will probably be the next stage in this process. The intention is to keep this as an ongoing project that students can join primarily in the summer but also during the academic year as time permits. Centre maintains an honors program named in honor of John C. Young in which students can propose research projects to be conducted during their senior year. The research papers are presented in a Spring symposium open to the entire community and published by the college. The existence of this continuing project makes it possible for students to propose an honors thesis which is substantive while at the same time having a reasonable horizon. Robotics offers a very suitable platform for interesting and useful experimentation. Robots as Projects in CS Courses Before the inclusion of robotics projects in the artificial intelligence course, it was a struggle to find assignments that enabled students to implement many of the ideas that were presented. Most projects were simply too big to implement in a semester. However, whether it is using Lejos to program Lego robots, or the use of pyro with simulators, there are now ways to make interesting but reasonable assignments in small worlds where students can exercise the techniques that are introduced in lecture. They can use Lego robots to implement a simple reactive agent early in the course and later an agent which makes use of a more complex decision structure and state to make its decisions. The use of pyro allowed us to implement the logical agent for the “wumpus world” problem found in the text by Russell and Norvig (Russell and Norvig, 2003). Robots are also useful in introductory courses. There is about a two week segment on robotics at the very end of the CS I course. Having learned Python in the course, the students have not had any experience with complex syntax or the need to declare variables and types. The use of NQC to program Lego robots gives them an opportunity to try a second language with a more complicated syntax as well as to engage some other ideas like asynchronous execution. One of the easiest tasks to program is simple random motion in which the robot moves forward for a random amount of time and then turns either left or right for a random amount of time. Once the students learn to use the light sensor to watch out for a black boundary line, a second task can be added to back up whenever the boundary is detected. Almost every student writes at least one incorrect solution to this problem. Unless they include code to stop the random walk behavior while the boundary avoidance task is running, they will occasionally see their robot plunge recklessly over the edge when the random walk behavior interrupts. It is a very good lesson on the difference between synchronous and asynchronous behavior. Other useful assignments include designing wall or line followers, agents which measure lengths or count stripes, and proximity sensors.